<!DOCTYPE html>
<html lang="zh-cn">
<head>
    <meta charset="utf-8" />
    
        <title>Hadoop Day 3 | X-Blog</title>
    

    <meta http-equiv="X-UA-Compatible" content="IE=Edge,chrome=1"/>
<meta name="viewport" content="width=device-width,initial-scale=1.0,maximum-scale=1.0,user-scalable=no"/>
<meta name="format-detection" content="telephone=no">
<meta name="author" content="[object Object]" />
<meta name="designer" content="minfive" />
<meta name="keywords" content="Xiang blog, 程序员, SCI论文, Python, Hadoop, 深度炼丹"/>
<meta name="description" content="日常学习与兴趣交流的个人博客" />

<meta name="apple-mobile-web-app-capable" content="yes" />
<meta name="apple-mobile-web-app-status-bar-style" content="black" />
<meta name="format-detection" content="telephone=yes" />
<meta name="mobile-web-app-capable" content="yes" />
<meta name="robots" content="all" />

<link rel="canonical" href="https://spaces-x.github.io/2018/08/19/hadoop-d-3/index.html">

<link rel="icon" type="image/png" href="https://www.easyicon.net/api/resizeApi.php?id=546977&size=32" sizes="32x32">














<!-- Prefetch -->






<!-- CSS -->
<link rel="stylesheet" href="/scss/base/index.css">

<!-- RSS -->
<link rel="alternate" href="/atom.xml" title="Space-X">

<!-- 统计 -->
<!-- 百度统计 -->


<!-- Global site tag (gtag.js) - Google Analytics -->


    
    
    <link rel="stylesheet" href="/scss/views/page/post.css">

</head>
<body ontouchstart>
    
        <!-- loading 页面 -->
<div id="page-loading" class="page page-loading" style="background-image: url('http://oo12ugek5.bkt.clouddn.com/blog/images/loader.gif')"></div>
    

    <div id="page" class="page js-hidden">
        
    <!-- 页头 -->
<header class="page__small-header page__header--small">
    <nav class="page__navbar">
        <div class="page__container navbar-container">
            <a class="page__logo" href="/" title="Space-X" alt="Space-X">
                <img src="https://s1.ax1x.com/2018/07/15/PMfvfP.png" alt="Space-X">
            </a>

            <nav class="page__nav">
                <ul class="nav__list clearfix">
                    
                        
                        <li class="nav__item">
                            <a href="/" alt="首页" title="首页">首页</a>
                        </li>
                    
                        
                        <li class="nav__item">
                            <a href="/archives" alt="归档" title="归档">归档</a>
                        </li>
                    
                        
                        <li class="nav__item">
                            <a href="/about" alt="关于" title="关于">关于</a>
                        </li>
                    
                </ul>
            </nav>

            <button class="page__menu-btn" type="button">
                <i class="iconfont icon-menu"></i>
            </button>
        </div>
    </nav>
</header>


        
    <main class="page__container page__main">
    <div class="page__content">
        <article class="page__post">
    <div class="post__cover">
        <img src="http://wx2.sinaimg.cn/mw690/0060lm7Tly1fuh930qfnrj30ic0ababp.jpeg" alt="Hadoop Day 3">
    </div>

    <header class="post__info">
        <h1 class="post__title">Hadoop Day 3</h1>

        <div class="post__mark">
            <div class="mark__block">
                <i class="mark__icon iconfont icon-write"></i>
                <ul class="mark__list clearfix">
                    <li class="mark__item">
                        <a href="https://www.github.com/spaces-x">WeiXiang</a>
                    </li>
                </ul>
            </div>
            
            <div class="mark__block">
                <i class="mark__icon iconfont icon-time"></i>
                <ul class="mark__list clearfix">
                    <li class="mark__item"><span>2018-08-19</span></li>
                </ul>
            </div>

            <div class="mark__block">
                <i class="mark__icon iconfont icon-tab"></i>
                <ul class="mark__list clearfix">
                    
                        <li class="mark__item">
                            <a href="/tags/hadoop/">Hadoop</a>
                        </li>
                    
                        <li class="mark__item">
                            <a href="/tags/读取数据/">读取数据</a>
                        </li>
                    
                </ul>
            </div>

            
        </div>
    </header>

    <div class="post__content">
        <h3 id="读取数据部分的关系图"><a href="#读取数据部分的关系图" class="headerlink" title="读取数据部分的关系图"></a>读取数据部分的关系图</h3><p><img src="https://7n.w3cschool.cn/attachments/image/wk/hadoop/mapreduce-inputformat.png" alt=""></p>
<h3 id="InputFormat"><a href="#InputFormat" class="headerlink" title="InputFormat"></a>InputFormat</h3><p>首先我们先看看官方文档对 <a href="http://hadoop.apache.org/docs/r2.7.2/api/org/apache/hadoop/mapreduce/InputFormat.html" target="_blank" rel="noopener">InputFormat</a>的解释</p>
<p>从api文档中我们可以了解到InputFormat主要干3件事:</p>
<ol>
<li>验证作业输入的正确性，如格式等</li>
<li>将输入文件切割成逻辑分片(InputSplit)，一个InputSplit将会被分配给一个独立的Map任务</li>
<li>提供RecordReader实现，读取InputSplit中的”K-V对”供Mapper使用</li>
</ol>
<p>基于文件的InputFormats（通常是FileInputFormat的子类）的默认行为是根据输入文件的总大小（以字节为单位）将输入拆分为逻辑InputSplits。 但是，输入文件的FileSystem块大小被视为输入拆分大小的上限。 可以通过mapreduce.input.file.inputformat.split.minsize设置拆分大小的下限。</p>
<p>显然，基于输入大小的逻辑分割对于许多应用来说是不够的，因为要尊守记录边界。 在这种情况下，应用程序还必须实现一个RecordReader，负责尊守记录边界，并将逻辑InputSplit的面向记录的视图呈现给单个任务。</p>
<p>方法：</p>
<ol>
<li>List getSplits(): 获取由输入文件计算出输入分片(InputSplit)，解决数据或文件分割成片问题。</li>
<li>RecordReader &lt;k,v&gt;createRecordReader():&lt;k,v&gt; 创建RecordReader，从InputSplit中读取数据，解决读取分片中数据问题</li>
</ol>
<p><strong>TextInputFormat:</strong> 输入文件中的每一行就是一个记录，Key是这一行的byte offset，而value是这一行的内容</p>
<p><strong>KeyValueTextInputFormat:</strong> 输入文件中每一行就是一个记录，第一个分隔符字符切分每行。在分隔符字符之前的内容为Key，在之后的为Value。分隔符变量通过key.value.separator.in.input.line变量设置，默认为(\t)字符。</p>
<p><strong>NLineInputFormat:</strong> 与TextInputFormat一样，但每个数据块必须保证有且只有Ｎ行，mapred.line.input.format.linespermap属性，默认为１</p>
<p><strong>SequenceFileInputFormat:</strong>  一个用来读取字符流数据的InputFormat，&lt;key,value&gt;为用户自定义的。字符流数据是Hadoop自定义的压缩的二进制数据格式。它用来优化从一个MapReduce任务的输出到另一个MapReduce任务的输入之间的数据传输过程。&lt;/key,value&gt;</p>
<h4 id="FileInputFormat"><a href="#FileInputFormat" class="headerlink" title="FileInputFormat"></a>FileInputFormat</h4><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">/** </span></span><br><span class="line"><span class="comment"> * A base class for file-based &#123;<span class="doctag">@link</span> InputFormat&#125;s.</span></span><br><span class="line"><span class="comment"> * </span></span><br><span class="line"><span class="comment"> * &lt;p&gt;&lt;code&gt;FileInputFormat&lt;/code&gt; is the base class for all file-based </span></span><br><span class="line"><span class="comment"> * &lt;code&gt;InputFormat&lt;/code&gt;s. This provides a generic implementation of</span></span><br><span class="line"><span class="comment"> * &#123;<span class="doctag">@link</span> #getSplits(JobContext)&#125;.</span></span><br><span class="line"><span class="comment"> * Subclasses of &lt;code&gt;FileInputFormat&lt;/code&gt; can also override the </span></span><br><span class="line"><span class="comment"> * &#123;<span class="doctag">@link</span> #isSplitable(JobContext, Path)&#125; method to ensure input-files are</span></span><br><span class="line"><span class="comment"> * not split-up and are processed as a whole by &#123;<span class="doctag">@link</span> Mapper&#125;s.</span></span><br><span class="line"><span class="comment"> */</span></span><br><span class="line"><span class="meta">@InterfaceAudience</span>.Public</span><br><span class="line"><span class="meta">@InterfaceStability</span>.Stable</span><br><span class="line"><span class="keyword">public</span> <span class="keyword">abstract</span> <span class="class"><span class="keyword">class</span> <span class="title">FileInputFormat</span>&lt;<span class="title">K</span>, <span class="title">V</span>&gt; <span class="keyword">extends</span> <span class="title">InputFormat</span>&lt;<span class="title">K</span>, <span class="title">V</span>&gt; </span>&#123;</span><br><span class="line">  <span class="keyword">public</span> <span class="keyword">static</span> <span class="keyword">final</span> String INPUT_DIR = </span><br><span class="line">    <span class="string">"mapreduce.input.fileinputformat.inputdir"</span>;                 <span class="comment">//输入路径配置名称</span></span><br><span class="line">  <span class="keyword">public</span> <span class="keyword">static</span> <span class="keyword">final</span> String SPLIT_MAXSIZE = </span><br><span class="line">    <span class="string">"mapreduce.input.fileinputformat.split.maxsize"</span>;            <span class="comment">// split大小的最大值配置名称</span></span><br><span class="line">  <span class="keyword">public</span> <span class="keyword">static</span> <span class="keyword">final</span> String SPLIT_MINSIZE = </span><br><span class="line">    <span class="string">"mapreduce.input.fileinputformat.split.minsize"</span>;            <span class="comment">//split大小的最小值配置名称</span></span><br><span class="line">  <span class="keyword">public</span> <span class="keyword">static</span> <span class="keyword">final</span> String PATHFILTER_CLASS = </span><br><span class="line">    <span class="string">"mapreduce.input.pathFilter.class"</span>;</span><br><span class="line">  <span class="keyword">public</span> <span class="keyword">static</span> <span class="keyword">final</span> String NUM_INPUT_FILES =</span><br><span class="line">    <span class="string">"mapreduce.input.fileinputformat.numinputfiles"</span>;</span><br><span class="line">  <span class="keyword">public</span> <span class="keyword">static</span> <span class="keyword">final</span> String INPUT_DIR_RECURSIVE =</span><br><span class="line">    <span class="string">"mapreduce.input.fileinputformat.input.dir.recursive"</span>;       <span class="comment">//是否递归dir的boolean的配置名称</span></span><br><span class="line">  <span class="keyword">public</span> <span class="keyword">static</span> <span class="keyword">final</span> String LIST_STATUS_NUM_THREADS = </span><br><span class="line">    <span class="string">"mapreduce.input.fileinputformat.list-status.num-threads"</span>;    <span class="comment">//收集filestatus的list时候的线程数目配置名称</span></span><br><span class="line">  <span class="keyword">public</span> <span class="keyword">static</span> <span class="keyword">final</span> <span class="keyword">int</span> DEFAULT_LIST_STATUS_NUM_THREADS = <span class="number">1</span>;    <span class="comment">// 默认使用一个线程来做list filestatus</span></span><br><span class="line"></span><br><span class="line">  <span class="keyword">private</span> <span class="keyword">static</span> <span class="keyword">final</span> Log LOG = LogFactory.getLog(FileInputFormat.class);</span><br><span class="line"></span><br><span class="line">  <span class="keyword">private</span> <span class="keyword">static</span> <span class="keyword">final</span> <span class="keyword">double</span> SPLIT_SLOP = <span class="number">1.1</span>;   <span class="comment">// 10% slop     //当文件大小少于SPLIT_SLOP*SPLIT_SIZE时不分割</span></span><br><span class="line">  </span><br><span class="line">  <span class="meta">@Deprecated</span></span><br><span class="line">  <span class="keyword">public</span> <span class="keyword">static</span> <span class="keyword">enum</span> Counter &#123; </span><br><span class="line">    BYTES_READ</span><br><span class="line">  &#125;</span><br><span class="line">  ...</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure>
<p>类的主要成员变量都是一些配置名称的String。下面我们从这个类的主要函数说起。</p>
<p>在List<filestatus> listStatus(JobContext job)中列出了输入的目录列表。子类可以对这个方法重载例如子类可以满足特定正则表达式的输入路径。</filestatus></p>
<blockquote>
<ul>
<li>@param job the job to list input paths for</li>
<li>@return array of FileStatus objects</li>
<li>@throws IOException if zero items.</li>
</ul>
</blockquote>
<p>listStatus函数通过JobContext来获取配置信息，通过读取配置信息进行判断来进一步建立InputFile的PathFilter，如果配置中numThreads==1则使用singleThreadedListStatus()函数来的到List<filestatus>，否则建立LocatedFileStatusFetcher对象，多线程地得到List<filestatus>。<br><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">protected</span> List&lt;FileStatus&gt; <span class="title">listStatus</span><span class="params">(JobContext job</span></span></span><br><span class="line"><span class="function"><span class="params">                                      )</span> <span class="keyword">throws</span> IOException </span>&#123;</span><br><span class="line">  Path[] dirs = getInputPaths(job);</span><br><span class="line">  <span class="keyword">if</span> (dirs.length == <span class="number">0</span>) &#123;</span><br><span class="line">    <span class="keyword">throw</span> <span class="keyword">new</span> IOException(<span class="string">"No input paths specified in job"</span>);</span><br><span class="line">  &#125;</span><br><span class="line">  </span><br><span class="line">  <span class="comment">// get tokens for all the required FileSystems..</span></span><br><span class="line">  TokenCache.obtainTokensForNamenodes(job.getCredentials(), dirs, </span><br><span class="line">                                      job.getConfiguration());</span><br><span class="line"></span><br><span class="line">  <span class="comment">// Whether we need to recursive look into the directory structure</span></span><br><span class="line">  <span class="keyword">boolean</span> recursive = getInputDirRecursive(job);</span><br><span class="line"></span><br><span class="line">  <span class="comment">// creates a MultiPathFilter with the hiddenFileFilter and the</span></span><br><span class="line">  <span class="comment">// user provided one (if any).</span></span><br><span class="line">  List&lt;PathFilter&gt; filters = <span class="keyword">new</span> ArrayList&lt;PathFilter&gt;();</span><br><span class="line">  filters.add(hiddenFileFilter);</span><br><span class="line">  PathFilter jobFilter = getInputPathFilter(job);</span><br><span class="line">  <span class="keyword">if</span> (jobFilter != <span class="keyword">null</span>) &#123;</span><br><span class="line">    filters.add(jobFilter);</span><br><span class="line">  &#125;</span><br><span class="line">  PathFilter inputFilter = <span class="keyword">new</span> MultiPathFilter(filters);</span><br><span class="line">  </span><br><span class="line">  List&lt;FileStatus&gt; result = <span class="keyword">null</span>;</span><br><span class="line"></span><br><span class="line">  <span class="keyword">int</span> numThreads = job.getConfiguration().getInt(LIST_STATUS_NUM_THREADS,</span><br><span class="line">      DEFAULT_LIST_STATUS_NUM_THREADS);        <span class="comment">//读取配置文件中的线程数</span></span><br><span class="line">  Stopwatch sw = <span class="keyword">new</span> Stopwatch().start();</span><br><span class="line">  <span class="keyword">if</span> (numThreads == <span class="number">1</span>) &#123;</span><br><span class="line">    result = singleThreadedListStatus(job, dirs, inputFilter, recursive); <span class="comment">//单线程执行ListStatus</span></span><br><span class="line">  &#125; <span class="keyword">else</span> &#123;</span><br><span class="line">    Iterable&lt;FileStatus&gt; locatedFiles = <span class="keyword">null</span>;</span><br><span class="line">    <span class="keyword">try</span> &#123;</span><br><span class="line">      LocatedFileStatusFetcher locatedFileStatusFetcher = <span class="keyword">new</span> LocatedFileStatusFetcher(</span><br><span class="line">          job.getConfiguration(), dirs, recursive, inputFilter, <span class="keyword">true</span>);     <span class="comment">// 多线程建立LocatedFileStatusFetcher 来多线程执行listfilestatus，在fetcher中通过 Executors.newFixedThreadPool()建立线程池</span></span><br><span class="line">      locatedFiles = locatedFileStatusFetcher.getFileStatuses();</span><br><span class="line">    &#125; <span class="keyword">catch</span> (InterruptedException e) &#123;</span><br><span class="line">      <span class="keyword">throw</span> <span class="keyword">new</span> IOException(<span class="string">"Interrupted while getting file statuses"</span>);</span><br><span class="line">    &#125;</span><br><span class="line">    result = Lists.newArrayList(locatedFiles);</span><br><span class="line">  &#125;</span><br><span class="line">  </span><br><span class="line">  sw.stop();</span><br><span class="line">  <span class="keyword">if</span> (LOG.isDebugEnabled()) &#123;</span><br><span class="line">    LOG.debug(<span class="string">"Time taken to get FileStatuses: "</span> + sw.elapsedMillis());</span><br><span class="line">  &#125;</span><br><span class="line">  LOG.info(<span class="string">"Total input paths to process : "</span> + result.size()); </span><br><span class="line">  <span class="keyword">return</span> result;</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure></filestatus></filestatus></p>
<p>getSplits() 用于将输入文件进行拆分成splits并返回 </p>
<blockquote>
<ul>
<li>Generate the list of files and make them into FileSplits.</li>
<li>@param job the job context</li>
<li>@throws IOException</li>
</ul>
</blockquote>
<p>通过配置文件来得到split的maxSize和minSize和filesystem的BlockSize，通过以上三个size可以算出 Math.max(minSize, Math.min(maxSize, blockSize))=splitsize。 通过上面的listStatus()来得到List<filestatus>并且变量list中的每一个FileStatus，按照每一个file的长度和splitsize来分片成splits</filestatus></p>
<figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">public</span> List&lt;InputSplit&gt; <span class="title">getSplits</span><span class="params">(JobContext job)</span> <span class="keyword">throws</span> IOException </span>&#123;</span><br><span class="line">   Stopwatch sw = <span class="keyword">new</span> Stopwatch().start();</span><br><span class="line">   <span class="keyword">long</span> minSize = Math.max(getFormatMinSplitSize(), getMinSplitSize(job));</span><br><span class="line">   <span class="keyword">long</span> maxSize = getMaxSplitSize(job);</span><br><span class="line"></span><br><span class="line">   <span class="comment">// generate splits</span></span><br><span class="line">   List&lt;InputSplit&gt; splits = <span class="keyword">new</span> ArrayList&lt;InputSplit&gt;();</span><br><span class="line">   List&lt;FileStatus&gt; files = listStatus(job);</span><br><span class="line">   <span class="keyword">for</span> (FileStatus file: files) &#123;</span><br><span class="line">     Path path = file.getPath();</span><br><span class="line">     <span class="keyword">long</span> length = file.getLen();</span><br><span class="line">     <span class="keyword">if</span> (length != <span class="number">0</span>) &#123;</span><br><span class="line">       BlockLocation[] blkLocations;</span><br><span class="line">       <span class="comment">// 得到blkLocations 要么通过LocatedFileStatus的getBlockLocation </span></span><br><span class="line">       <span class="comment">// 要么通过FileSystem的getFileBlockLocations</span></span><br><span class="line">       <span class="keyword">if</span> (file <span class="keyword">instanceof</span> LocatedFileStatus) &#123;</span><br><span class="line">         blkLocations = ((LocatedFileStatus) file).getBlockLocations();</span><br><span class="line">       &#125; <span class="keyword">else</span> &#123;</span><br><span class="line">         FileSystem fs = path.getFileSystem(job.getConfiguration());</span><br><span class="line">         blkLocations = fs.getFileBlockLocations(file, <span class="number">0</span>, length);</span><br><span class="line">       &#125;</span><br><span class="line">       <span class="keyword">if</span> (isSplitable(job, path)) &#123;</span><br><span class="line">         <span class="keyword">long</span> blockSize = file.getBlockSize();</span><br><span class="line">         <span class="keyword">long</span> splitSize = computeSplitSize(blockSize, minSize, maxSize);</span><br><span class="line"></span><br><span class="line">         <span class="keyword">long</span> bytesRemaining = length;</span><br><span class="line">         <span class="comment">// 对当前文件按照splitsize进行分割，分割成一个个split</span></span><br><span class="line">         <span class="keyword">while</span> (((<span class="keyword">double</span>) bytesRemaining)/splitSize &gt; SPLIT_SLOP) &#123;</span><br><span class="line">           <span class="keyword">int</span> blkIndex = getBlockIndex(blkLocations, length-bytesRemaining);  <span class="comment">//得到block的index</span></span><br><span class="line">           splits.add(makeSplit(path, length-bytesRemaining, splitSize,</span><br><span class="line">                       blkLocations[blkIndex].getHosts(),</span><br><span class="line">                       blkLocations[blkIndex].getCachedHosts()));</span><br><span class="line">           bytesRemaining -= splitSize;</span><br><span class="line">         &#125;</span><br><span class="line"></span><br><span class="line">         <span class="keyword">if</span> (bytesRemaining != <span class="number">0</span>) &#123;</span><br><span class="line">           <span class="keyword">int</span> blkIndex = getBlockIndex(blkLocations, length-bytesRemaining);</span><br><span class="line">           splits.add(makeSplit(path, length-bytesRemaining, bytesRemaining,</span><br><span class="line">                      blkLocations[blkIndex].getHosts(),</span><br><span class="line">                      blkLocations[blkIndex].getCachedHosts()));</span><br><span class="line">         &#125;</span><br><span class="line">       &#125; <span class="keyword">else</span> &#123; <span class="comment">// not splitable 不可分 整个文件作为split</span></span><br><span class="line">         splits.add(makeSplit(path, <span class="number">0</span>, length, blkLocations[<span class="number">0</span>].getHosts(),</span><br><span class="line">                     blkLocations[<span class="number">0</span>].getCachedHosts()));</span><br><span class="line">       &#125;</span><br><span class="line">     &#125; <span class="keyword">else</span> &#123; </span><br><span class="line">       <span class="comment">//Create empty hosts array for zero length files</span></span><br><span class="line">       splits.add(makeSplit(path, <span class="number">0</span>, length, <span class="keyword">new</span> String[<span class="number">0</span>]));</span><br><span class="line">     &#125;</span><br><span class="line">   &#125;</span><br><span class="line">   <span class="comment">// Save the number of input files for metrics/loadgen</span></span><br><span class="line">   job.getConfiguration().setLong(NUM_INPUT_FILES, files.size());</span><br><span class="line">   sw.stop();</span><br><span class="line">   <span class="keyword">if</span> (LOG.isDebugEnabled()) &#123;</span><br><span class="line">     LOG.debug(<span class="string">"Total # of splits generated by getSplits: "</span> + splits.size()</span><br><span class="line">         + <span class="string">", TimeTaken: "</span> + sw.elapsedMillis());</span><br><span class="line">   &#125;</span><br><span class="line">   <span class="keyword">return</span> splits;</span><br><span class="line"> &#125;</span><br></pre></td></tr></table></figure>
<p>制作切片split<br><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line"> <span class="comment">/*</span></span><br><span class="line"><span class="comment"> * 制作切片split，makeSplit调用FileSplit()函数来制作切片</span></span><br><span class="line"><span class="comment"> */</span></span><br><span class="line"><span class="function"><span class="keyword">protected</span> FileSplit <span class="title">makeSplit</span><span class="params">(Path file, <span class="keyword">long</span> start, <span class="keyword">long</span> length, </span></span></span><br><span class="line"><span class="function"><span class="params">                               String[] hosts, String[] inMemoryHosts)</span> </span>&#123;</span><br><span class="line">   <span class="keyword">return</span> <span class="keyword">new</span> FileSplit(file, start, length, hosts, inMemoryHosts);</span><br><span class="line"> &#125;</span><br></pre></td></tr></table></figure></p>
<p>对file进行切片，并附带cached-blocks信息<br><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br></pre></td><td class="code"><pre><span class="line"> <span class="comment">/** Constructs a split with host and cached-blocks information</span></span><br><span class="line"><span class="comment"> *</span></span><br><span class="line"><span class="comment"> * <span class="doctag">@param</span> file the file name</span></span><br><span class="line"><span class="comment"> * <span class="doctag">@param</span> start the position of the first byte in the file to process</span></span><br><span class="line"><span class="comment"> * <span class="doctag">@param</span> length the number of bytes in the file to process</span></span><br><span class="line"><span class="comment"> * <span class="doctag">@param</span> hosts the list of hosts containing the block</span></span><br><span class="line"><span class="comment"> * <span class="doctag">@param</span> inMemoryHosts the list of hosts containing the block in memory</span></span><br><span class="line"><span class="comment"> */</span></span><br><span class="line"><span class="function"><span class="keyword">public</span> <span class="title">FileSplit</span><span class="params">(Path file, <span class="keyword">long</span> start, <span class="keyword">long</span> length, String[] hosts,</span></span></span><br><span class="line"><span class="function"><span class="params">    String[] inMemoryHosts)</span> </span>&#123;</span><br><span class="line">  <span class="keyword">this</span>(file, start, length, hosts);</span><br><span class="line">  hostInfos = <span class="keyword">new</span> SplitLocationInfo[hosts.length];</span><br><span class="line">  <span class="keyword">for</span> (<span class="keyword">int</span> i = <span class="number">0</span>; i &lt; hosts.length; i++) &#123;</span><br><span class="line">    <span class="comment">// because N will be tiny, scanning is probably faster than a HashSet</span></span><br><span class="line">    <span class="keyword">boolean</span> inMemory = <span class="keyword">false</span>;</span><br><span class="line">    <span class="keyword">for</span> (String inMemoryHost : inMemoryHosts) &#123;</span><br><span class="line">      <span class="keyword">if</span> (inMemoryHost.equals(hosts[i])) &#123;</span><br><span class="line">        inMemory = <span class="keyword">true</span>;</span><br><span class="line">        <span class="keyword">break</span>;</span><br><span class="line">      &#125;</span><br><span class="line">    &#125;</span><br><span class="line">    hostInfos[i] = <span class="keyword">new</span> SplitLocationInfo(hosts[i], inMemory);</span><br><span class="line">  &#125;</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure></p>
<p>LineRecorder: 负责记录行的读取情况</p>
<blockquote>
<ul>
<li>param Configuration, job configuartion.</li>
<li>param FileSplit, split to read.</li>
<li>param byte[], delimiter bytes.</li>
</ul>
</blockquote>
<p>通过job的configuration获取文件系统fs并且以输入流的形式打开分片所在的文件，后续的逻辑是判断输入流输入的文件是否为压缩，如果是压缩的话解压后重新定位start和end。如果不是压缩的的话很简单，直接通过读取split的start和end作为LineRecordReader的start和end，并且将文件定位到分片的start处。在以上逻辑完成后判断当前分片是不是文件中的第一个分片(start==0?)如果不是的话则越过第一行。<br>为什么非开头的分片要越过第一行？这是由于按照blocksize大小读取的分片很可能不是按行对其的，而LineRecorderReader要处理的是行 保证行对齐是关键，通过忽略非开头分片的第一行可以做到行对其，其结果如下图所示</p>
<p><img src="https://7n.w3cschool.cn/attachments/image/wk/hadoop/mapreduce-split.png" alt="行对齐"></p>
<p>那么如何计算对于非开头分片应该忽略多少才能保证行对其，即计算非开头分片start的后移量是多少呢？ 主要是通过readLine()函数,该还函数返回到行末的偏移量。 后续分析。</p>
<figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">public</span> <span class="title">LineRecordReader</span><span class="params">(Configuration job, FileSplit split,</span></span></span><br><span class="line"><span class="function"><span class="params">    <span class="keyword">byte</span>[] recordDelimiter)</span> <span class="keyword">throws</span> IOException </span>&#123;</span><br><span class="line">  <span class="keyword">this</span>.maxLineLength = job.getInt(org.apache.hadoop.mapreduce.lib.input.</span><br><span class="line">    LineRecordReader.MAX_LINE_LENGTH, Integer.MAX_VALUE);</span><br><span class="line">  start = split.getStart();</span><br><span class="line">  end = start + split.getLength();</span><br><span class="line">  <span class="keyword">final</span> Path file = split.getPath();</span><br><span class="line">  compressionCodecs = <span class="keyword">new</span> CompressionCodecFactory(job);</span><br><span class="line">  codec = compressionCodecs.getCodec(file);</span><br><span class="line"></span><br><span class="line">  <span class="comment">// open the file and seek to the start of the split</span></span><br><span class="line">  <span class="keyword">final</span> FileSystem fs = file.getFileSystem(job);</span><br><span class="line">  fileIn = fs.open(file);</span><br><span class="line">  <span class="keyword">if</span> (isCompressedInput()) &#123;</span><br><span class="line">    decompressor = CodecPool.getDecompressor(codec);</span><br><span class="line">    <span class="keyword">if</span> (codec <span class="keyword">instanceof</span> SplittableCompressionCodec) &#123;</span><br><span class="line">      <span class="keyword">final</span> SplitCompressionInputStream cIn =</span><br><span class="line">        ((SplittableCompressionCodec)codec).createInputStream(</span><br><span class="line">          fileIn, decompressor, start, end,</span><br><span class="line">          SplittableCompressionCodec.READ_MODE.BYBLOCK);</span><br><span class="line">      in = <span class="keyword">new</span> CompressedSplitLineReader(cIn, job, recordDelimiter);</span><br><span class="line">      start = cIn.getAdjustedStart();</span><br><span class="line">      end = cIn.getAdjustedEnd();</span><br><span class="line">      filePosition = cIn; <span class="comment">// take pos from compressed stream</span></span><br><span class="line">    &#125; <span class="keyword">else</span> &#123;</span><br><span class="line">      in = <span class="keyword">new</span> SplitLineReader(codec.createInputStream(fileIn,</span><br><span class="line">          decompressor), job, recordDelimiter);</span><br><span class="line">      filePosition = fileIn;</span><br><span class="line">    &#125;</span><br><span class="line">  &#125; <span class="keyword">else</span> &#123;</span><br><span class="line">    fileIn.seek(start);</span><br><span class="line">    in = <span class="keyword">new</span> UncompressedSplitLineReader(</span><br><span class="line">        fileIn, job, recordDelimiter, split.getLength());</span><br><span class="line">    filePosition = fileIn;</span><br><span class="line">  &#125;</span><br><span class="line">  <span class="comment">// If this is not the first split, we always throw away first record</span></span><br><span class="line">  <span class="comment">// because we always (except the last split) read one extra line in</span></span><br><span class="line">  <span class="comment">// next() method.</span></span><br><span class="line">  <span class="keyword">if</span> (start != <span class="number">0</span>) &#123;              <span class="comment">// 如果不是文件中的第一个分片</span></span><br><span class="line">    start += in.readLine(<span class="keyword">new</span> Text(), <span class="number">0</span>, maxBytesToConsume(start));    <span class="comment">//忽略文件中的第一行 start后移。这里调用了readLine来计算后移的偏量</span></span><br><span class="line">  &#125;</span><br><span class="line">  <span class="keyword">this</span>.pos = start;</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure>
<p>接下来我们看看readLine函数,它主要通过判断是否设定分隔符来返回 自定义和默认方式的readline。<br><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">public</span> <span class="keyword">int</span> <span class="title">readLine</span><span class="params">(Text str, <span class="keyword">int</span> maxLineLength,</span></span></span><br><span class="line"><span class="function"><span class="params">                    <span class="keyword">int</span> maxBytesToConsume)</span> <span class="keyword">throws</span> IOException </span>&#123;</span><br><span class="line">  <span class="keyword">if</span> (<span class="keyword">this</span>.recordDelimiterBytes != <span class="keyword">null</span>) &#123;     <span class="comment">//分隔符数组为空？ 即未设定分隔符？</span></span><br><span class="line">    <span class="keyword">return</span> readCustomLine(str, maxLineLength, maxBytesToConsume);  <span class="comment">//自定义分隔符！</span></span><br><span class="line">  &#125; <span class="keyword">else</span> &#123;                                     </span><br><span class="line">    <span class="keyword">return</span> readDefaultLine(str, maxLineLength, maxBytesToConsume); <span class="comment">//否则使用默认的分隔符 即 '\r' or '\n'</span></span><br><span class="line">  &#125;</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure></p>


        
        <div class="post-announce">
            感谢您的阅读，本文由
            <a href="https://spaces-x.github.io">Space-X</a>
            版权所有。如若转载，请注明出处：Space-X（<a href="https://spaces-x.github.io/2018/08/19/hadoop-d-3/">https://spaces-x.github.io/2018/08/19/hadoop-d-3/</a>）
        </div>
        

        <div class="post__prevs">
            <div class="post__prev">
                
                <a href="/2018/08/08/hbase/" title="HBase"><i class="iconfont icon-prev"></i>HBase</a>
                
            </div>
            <div class="post__prev post__prev--right">
                
                <a href="/2019/01/18/nanjing/" title="Nanjing">Nanjing<i class="iconfont icon-next"></i></a>
                
            </div>
        </div>
    </div>
</article>

        
        
    </div>

    <aside class="page__sidebar">
    <!-- 
        <div class="sidebar__img">
            <img src="https://s1.ax1x.com/2018/08/08/PsWmg1.jpg" alt="Space-X" title="Space-X">
        </div>
     -->

    <form id="page-search-from" class="page__search-from" action="/search/">
        <label class="search-form__item">
            <input class="input" type="text" name="search" placeholder="Search...">
            <i class="iconfont icon-search"></i>
        </label>
    </form>

    
        <div class="sidebar__block">
            <h3 class="block__title">简介</h3>
            <p class="block__text">日常学习与兴趣交流的个人博客</p>
        </div>
    

    <div class="sidebar__block">
        <h3 class="block__title">文章分类</h3>
        <ul class="block-list"><li class="block-list-item"><a class="block-list-link" href="/categories/计算机网络/">计算机网络</a><span class="block-list-count">1</span></li><li class="block-list-item"><a class="block-list-link" href="/categories/算法/">算法</a><span class="block-list-count">1</span></li><li class="block-list-item"><a class="block-list-link" href="/categories/游玩/">游玩</a><span class="block-list-count">1</span></li><li class="block-list-item"><a class="block-list-link" href="/categories/welcome/">welcome</a><span class="block-list-count">1</span></li><li class="block-list-item"><a class="block-list-link" href="/categories/java设计模式/">java设计模式</a><span class="block-list-count">1</span></li><li class="block-list-item"><a class="block-list-link" href="/categories/hadoop/">hadoop</a><span class="block-list-count">1</span></li><li class="block-list-item"><a class="block-list-link" href="/categories/RPC/">RPC</a><span class="block-list-count">1</span><ul class="block-list-child"><li class="block-list-item"><a class="block-list-link" href="/categories/RPC/Hadoop/">Hadoop</a><span class="block-list-count">1</span></li></ul></li><li class="block-list-item"><a class="block-list-link" href="/categories/OS/">OS</a><span class="block-list-count">1</span></li><li class="block-list-item"><a class="block-list-link" href="/categories/Hbase/">Hbase</a><span class="block-list-count">1</span></li><li class="block-list-item"><a class="block-list-link" href="/categories/Hadoop/">Hadoop</a><span class="block-list-count">2</span></li><li class="block-list-item"><a class="block-list-link" href="/categories/Git-版本管理/">Git 版本管理</a><span class="block-list-count">1</span></li><li class="block-list-item"><a class="block-list-link" href="/categories/GFW/">GFW</a><span class="block-list-count">1</span></li></ul>
    </div>
    
    <div class="sidebar__block">
        <h3 class="block__title">最新文章</h3>
        
        <ul class="block-list latest-post-list">
            
                    <li class="latest-post-item">
                        <a href="/2019/05/10/hadoopRPC/" title="hadoopRPC">
                            <div class="item__cover">
                                <img src="https://s2.ax1x.com/2019/05/10/EWAoB6.md.jpg" alt="hadoopRPC" />
                            </div>
                            <div class="item__info">
                                <h3 class="item__title">hadoopRPC</h3>
                                <span class="item__text">2019-05-10</span>
                            </div>
                        </a>
                    </li>
                
                    <li class="latest-post-item">
                        <a href="/2019/04/04/memorypaging/" title="Memory Paging">
                            <div class="item__cover">
                                <img src="https://s2.ax1x.com/2019/04/04/A2ifUO.md.jpg" alt="Memory Paging" />
                            </div>
                            <div class="item__info">
                                <h3 class="item__title">Memory Paging</h3>
                                <span class="item__text">2019-04-04</span>
                            </div>
                        </a>
                    </li>
                
                    <li class="latest-post-item">
                        <a href="/2019/03/31/DecoratorMode/" title="DecoratorMode">
                            <div class="item__cover">
                                <img src="https://s2.ax1x.com/2019/03/31/ArQrQJ.md.jpg" alt="DecoratorMode" />
                            </div>
                            <div class="item__info">
                                <h3 class="item__title">DecoratorMode</h3>
                                <span class="item__text">2019-03-31</span>
                            </div>
                        </a>
                    </li>
                
                    <li class="latest-post-item">
                        <a href="/2019/01/18/alogorithm/" title="Algorithm1">
                            <div class="item__cover">
                                <img src="https://s2.ax1x.com/2019/01/18/k9gwCD.jpg" alt="Algorithm1" />
                            </div>
                            <div class="item__info">
                                <h3 class="item__title">Algorithm1</h3>
                                <span class="item__text">2019-01-18</span>
                            </div>
                        </a>
                    </li>
                
        </ul>
    
    </div>

    <div class="sidebar__block">
        <h3 class="block__title">文章标签</h3>
        
        <ul class="block-list tag-list clearfix">
            
                    <li class="tag-item">
                        <a class="tag-link" href="/tags/Eclipse/">Eclipse</a>
                    </li>  
                
                    <li class="tag-item">
                        <a class="tag-link" href="/tags/Google-PageRank/">Google,PageRank</a>
                    </li>  
                
                    <li class="tag-item">
                        <a class="tag-link" href="/tags/Great-Firewall/">Great Firewall</a>
                    </li>  
                
                    <li class="tag-item">
                        <a class="tag-link" href="/tags/Hadoop/">Hadoop</a>
                    </li>  
                
                    <li class="tag-item">
                        <a class="tag-link" href="/tags/Hadoop部署/">Hadoop部署</a>
                    </li>  
                
                    <li class="tag-item">
                        <a class="tag-link" href="/tags/Hbase/">Hbase</a>
                    </li>  
                
                    <li class="tag-item">
                        <a class="tag-link" href="/tags/Learning/">Learning</a>
                    </li>  
                
                    <li class="tag-item">
                        <a class="tag-link" href="/tags/Life/">Life</a>
                    </li>  
                
                    <li class="tag-item">
                        <a class="tag-link" href="/tags/Map-Reduce/">Map-Reduce</a>
                    </li>  
                
                    <li class="tag-item">
                        <a class="tag-link" href="/tags/RPC/">RPC</a>
                    </li>  
                
                    <li class="tag-item">
                        <a class="tag-link" href="/tags/TCP-Socket通信/">TCP,Socket通信</a>
                    </li>  
                
                    <li class="tag-item">
                        <a class="tag-link" href="/tags/git/">git</a>
                    </li>  
                
                    <li class="tag-item">
                        <a class="tag-link" href="/tags/hadoop/">hadoop</a>
                    </li>  
                
                    <li class="tag-item">
                        <a class="tag-link" href="/tags/hobbies/">hobbies</a>
                    </li>  
                
                    <li class="tag-item">
                        <a class="tag-link" href="/tags/java/">java</a>
                    </li>  
                
                    <li class="tag-item">
                        <a class="tag-link" href="/tags/linux/">linux</a>
                    </li>  
                
                    <li class="tag-item">
                        <a class="tag-link" href="/tags/love/">love</a>
                    </li>  
                
                    <li class="tag-item">
                        <a class="tag-link" href="/tags/memory/">memory</a>
                    </li>  
                
                    <li class="tag-item">
                        <a class="tag-link" href="/tags/noSQL/">noSQL</a>
                    </li>  
                
                    <li class="tag-item">
                        <a class="tag-link" href="/tags/paging/">paging</a>
                    </li>  
                
                    <li class="tag-item">
                        <a class="tag-link" href="/tags/shadowsocks/">shadowsocks</a>
                    </li>  
                
                    <li class="tag-item">
                        <a class="tag-link" href="/tags/travel/">travel</a>
                    </li>  
                
                    <li class="tag-item">
                        <a class="tag-link" href="/tags/分支定界/">分支定界</a>
                    </li>  
                
                    <li class="tag-item">
                        <a class="tag-link" href="/tags/列式数据库/">列式数据库</a>
                    </li>  
                
                    <li class="tag-item">
                        <a class="tag-link" href="/tags/动态规划/">动态规划</a>
                    </li>  
                
                    <li class="tag-item">
                        <a class="tag-link" href="/tags/版本管理/">版本管理</a>
                    </li>  
                
                    <li class="tag-item">
                        <a class="tag-link" href="/tags/装饰器模式/">装饰器模式</a>
                    </li>  
                
                    <li class="tag-item">
                        <a class="tag-link" href="/tags/计算机网络/">计算机网络</a>
                    </li>  
                
                    <li class="tag-item">
                        <a class="tag-link" href="/tags/读取数据/">读取数据</a>
                    </li>  
                
        </ul>
    
    </div>

    <!-- <div class="sidebar__block">
        <h3 class="block__title">友情链接</h3>
        <ul class="block-list">
            
        </ul>
    </div> -->
</aside>
</main>


        
            <!-- 页脚 -->
<footer class="page__footer">
    <section class="footer__top">
        <div class="page__container footer__container">
            
            <div class="footer-top__item footer-top__item--2">
                <h3 class="item__title">关于</h3>
                <div class="item__content">
                    <p class="item__text">本站是基于 Hexo 搭建的静态资源博客，主要用于分享日常学习、生活及工作的一些心得总结，欢迎点击右下角订阅 rss。</p>
                    <ul class="footer__contact-info">
                        <li class="contact-info__item">
                            <i class="iconfont icon-address"></i>
                            <span>Dalian, Lianning Province, China</span>
                        </li>
                        <li class="contact-info__item">
                            <i class="iconfont icon-email2"></i>
                            <span>dlut.weixiang@gmail.com</span>
                        </li>
                    </ul>
                </div>
            </div>

            
            
                <div class="footer-top__item footer__image">
                    <img src="https://s1.ax1x.com/2018/08/08/PsWmg1.jpg" alt="logo" title="Space-X">
                </div>
            
            
            
            
                
                    <div class="footer-top__item">
                        <h3 class="item__title">友情链接</h3>
                        <div class="item__content">
                            <ul class="footer-top__list">
                                
                                    <li class="list-item">
                                        <a href="https://github.com/Mrminfive/hexo-theme-skapp" title="hexo-theme-skapp" target="_blank">hexo-theme-skapp</a>
                                    </li>
                                
                            </ul>
                        </div>
                    </div>
                
                    <div class="footer-top__item">
                        <h3 class="item__title">构建工具</h3>
                        <div class="item__content">
                            <ul class="footer-top__list">
                                
                                    <li class="list-item">
                                        <a href="https://hexo.io/" title="Blog Framework" target="_blank">Hexo</a>
                                    </li>
                                
                            </ul>
                        </div>
                    </div>
                
            
        </div>
    </section>
    <section class="footer__bottom">
        <div class="page__container footer__container">
            <p class="footer__copyright">©
                <a href="https://github.com/Mrminfive/hexo-theme-skapp" target="_blank">Skapp</a> 2017 powered by
                <a href="http://hexo.io/" target="_blank">Hexo</a>, made by 
                <a href="https://github.com/Mrminfive" target="_blank">minfive</a>.
            </p>
            <ul class="footer__social-network clearfix">
                
                
                    <li class="social-network__item">
                        <a href="https://github.com/spaces-x" target="_blank" title="github">
                            <i class="iconfont icon-github"></i>
                        </a>
                    </li>
                
                    <li class="social-network__item">
                        <a href="mailto:dlut.weixiang@gmail.com" target="_blank" title="email">
                            <i class="iconfont icon-email"></i>
                        </a>
                    </li>
                
                    <li class="social-network__item">
                        <a href="/atom.xml" target="_blank" title="rss">
                            <i class="iconfont icon-rss"></i>
                        </a>
                    </li>
                
                
            </ul>
        </div>
    </section>
</footer>
        

        
            <!-- 返回顶部 -->
<div id="back-top" class="back-top back-top--hidden js-hidden">
    <i class="iconfont icon-top"></i>
</div>
        
    </div>

    <!-- build:js /js/common.js -->
        <script type="text/javascript" src="js/common/utils.js"></script>
        <script type="text/javascript" src="js/common/pack.js"></script>
        <script type="text/javascript" src="js/common/animation.js"></script>
        <script type="text/javascript" src="js/layout/loading.js"></script>
        <script type="text/javascript" src="js/layout/header.js"></script>
        <script type="text/javascript" src="js/layout/back-top.js"></script>
        <script type="text/javascript" src="js/layout/post.js"></script>
    <!-- endbuild -->

    
    <script src="/js/page/post.js"></script>


    
    



    <!-- 不蒜子统计 -->

<script async src="//dn-lbstatics.qbox.me/busuanzi/2.3/busuanzi.pure.mini.js">
</script>


     








</body>
</html>